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DETAILED ACTION 



1 . This communication is responsive to the Amendment filed February 1 0, 2003. 

2. Claims 1-30, 32-38, 40-44 and 46-56 are pending in this application. Claims 1, 
1 1 , 19, 23, 30, 32, 41 , 46, 48, 50 and 55 are independent claims. In the Amendment, 
claims 51-56 were added, claims 31, 39 and 45 were cancelled, and claims 5, 6, 19, 27, 
28, 32, 33, 35, 36, 38, and 40-43 were amended. This action is non-final. 



The following is a quotation of the second paragraph of 35 U.S.C. 112: 

The specification shall conclude with one or more claims particularly pointing out and distinctly 
claiming the subject matter which the applicant regards as his invention. 

3. Claims 32-38, 55 and 56 are rejected under 35 U.S.C. 112, second paragraph, 

as being indefinite for failing to particularly point out and distinctly claim the subject 

matter which applicant regards as the invention. 

Claim 32 recites the limitation "the set of search criteria" in the third line of the 
claim. There is insufficient antecedent basis for this limitation in the claim. 

In the interest of compact prosecution, the examiner assumes that "the" should 
be replaced with 'a' such that the limitation reads "a set of search criteria". 

Claims 33-38 are each dependent upon claim 32, and are therefore indefinite for 
the same reason as claim 32. 

Claim 35 recites the limitation "the initial search criteria" in the first line of the 
claim. There is insufficient antecedent basis for this limitation in the claim. 



Claim Rejections - 35 USC §112 
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In the interest of compact prosecution, the examiner assumes that "the" should 
be replaced with 'an' such that the limitation reads "an initial search criteria". 

Claim 55 recites the limitation "the user feedback" in the fifth line of the claim. 
There is insufficient antecedent basis for this limitation in the claim. The previous line of 
the claim recites, "receiving feedback..." which does not explicitly involve a user as 
written. Therefore, consistency needs to be brought to these limitations. 

Claim 56 recites the limitation "the receiving user feedback" in the second line of 
the claim. There is insufficient antecedent basis for this limitation in the claim for the 
same reasons as above. 

In the interest of compact prosecution, the examiner ignores the word "user" in 
both of these limitations (claims 55 and 56) to take the broadest reasonable 
interpretation of the claims. 

Claim Rejections - 35 USC § 102 

The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that 
form the basis for the rejections under this section made in this Office action: 
A person shall be entitled to a patent unless - 

(a) the invention was known or used by others in this country, or patented or described in a printed 
publication in this or a foreign country, before the invention thereof by the applicant for a patent. 

(b) the invention was patented or described in a printed publication in this or a foreign country or in public 
use or on sale in this country, more than one year prior to the date of application for patent in the United 
States. 

4. Claims 1 , 8-9, 32, 35-38, 40-44 and 48-51 are rejected under 35 U.S.C. 1 02(b) 
as being clearly anticipated by the article entitled "Relevance Feedback Techniques in 
Interactive Content-Based Image Retrieval" by Rui et al. (hereinafter "RFT") 
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Referring to claim 1, RFT discloses the invention exactly as claimed. See 
sections 3 and 4 on pages 4-8 of the article for this disclosure. Refer specifically to the 
method on page 4 and Figure 2 for the details of this disclosure. In particular, RFT 
teaches "one or more computer readable media having stored thereon a plurality of 
instructions [See section 4] that, when executed by one or more processors, causes the 
one or more processors to perform acts including: 

receiving [Step 2] an initial image [query object Q] selection; 

generating [Steps 2-3] a plurality of query vectors [Q's feature vectors f] by 
extracting, for each query vector, one of a plurality of low-level features [representations 
r] from the initial image selection [See Figure 2]; 

selecting [Steps 4-7] a set of potentially relevant images [most similar objects] 
based at least in part on distances [S(//)J between the plurality of query vectors [Q's 
feature vectors f] and a plurality of feature vectors [O's feature vectors f] corresponding 
to low-level features of a plurality of images [See Figure 2]; 

receiving feedback [Step 8] regarding the relevance of one or more images of the 
set of potentially relevant images; 

generating [Step 9] a new plurality of query vectors [adjusted Q] based at least in 
part on the feedback [See section 3.2]; 

generating [Step 9] a weighting of feature elements [updates the weights] based 
at least in part on the feedback [See section 3.1]; and 

selecting [Steps 2-7 (after step 10)] a new set of potentially relevant images 
based at least in part on both the weighting of feature elements and distances between 
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the new plurality of query vectors and the plurality of feature vectors [See step 10]" as 
claimed. 

Referring to claim 8, RFT discloses the computer readable media as claimed. 
See steps 4-6 of RFT's method in light of sections 3.1 and 3.2 for this disclosure. In 
particular, RFT's weight [WJ for each of a plurality [i] of distances [S(fj)] between a query 
vector and a corresponding feature vector is calculated as claimed. That is, the claimed 
equation is equivalent to equations (6) - (8) in the RFT reference as derived for Wj. 

Referring to claim 9, RFT discloses the computer readable media as claimed. 
See steps 8 & 9 of RFT's method for this disclosure. In particular, RFT teaches the 
computer readable media as recited in claim 1, as above, "wherein the receiving 
feedback [Step 8] comprises receiving feedback from a user ['the user marks it as highly 
relevant...' (see steps 8 and 9)]" as claimed. 

Referring to claim 32, RFT discloses the image retrieval method as claimed. See 
the discussion of claim 1 above and the pertinent portions of the RFT reference for this 
disclosure. In particular, RFT teaches a method comprising: 

"for one of a plurality of images [Objects 0] and each of a plurality of features [f], 

generating [Steps 2-3], based on a set of search criteria [Query Object Q], a 
query vector [Q's feature vector f (See Figure 2)] for the feature [one of features U - fj, 

identifying a feature vector [f], corresponding to the image [O], for the feature 
[one of features U - i], and 

determining [Steps 4 & 5] how closely [S(/))]the feature vector matches the query 
vector; and 
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determining [Step 6] how closely the image [O] matches the set of search criteria 
[Q] based on how closely [S], for the plurality of features, the feature vectors match the 
query vectors, wherein generating the query vector comprises generating the query 
vector based at least in part on user relevance feedback regarding how relevant images 
previously displayed to a user were [See Steps 8-10 and the discussion regarding claim 
1 above]" as claimed. 

Referring to claim 35, RFT discloses the method exactly as claimed. See the 
discussion regarding claims 1 & 32 above for the details of this disclosure. Initial search 
criteria for RFT's system comprises an image (Query Object Q : See section 2) as 
claimed. 

Referring to claims 36 and 37, RFT discloses the method exactly as claimed. 
See the discussion regarding claims 1 & 32 above for the details of this disclosure. 
RFT's determining how closely the images match (Steps 4-6) performs exactly as 
claimed in claim 36, while the weighted (by weighting matrix W) summation is calculated 
based on user relevance feedback (See Steps 8-10 and Sections 3.1 - 3.2) as claimed 
in claim 37. 

Claim 38 is rejected on the same basis as claim 32. See the discussions 
regarding claims 1 and 32 above for the details of this disclosure. 

Claims 40-42 are rejected on the same basis as claim 32. See the discussions 
regarding claims 1 and 32 above for the details of this disclosure. 
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Claims 43 and 44 are rejected on the same basis as claims 36 and 37 
respectively, in light of the basis for claim 40 above. See the discussions regarding 
claims 1, 32, 36-37 and 40 for the details of this disclosure. 

Claims 48 and 49 are rejected on the same basis as claim 8 above. See the 
discussions regarding claims 1 and 8 for the details of this disclosure. 

Claim 50 is rejected on the same basis as claim 1 above. See the discussion 
regarding claim 1 above, and Section 4 of the RFT reference for the details of this 
disclosure. In particular, RFT teaches "a system comprising: a client device [user 
computer]; a collection of a plurality of images [image database]; and an image server 
[MARS system], coupled to the client device and the collection of a plurality of images 
[accessible via the internet], the image server to receive image retrieval request from 
the client device and to... [perform the method as discussed with regard to claim 1 
above]" as claimed. 

Referring to claim 51 , RFT discloses the computer readable media exactly as 
claimed. See step 8 of RFT's method and section 3.1 of the RFT reference regarding 
the relevance scores for this disclosure. In particular, RFT's "receiving feedback [Step 
8] comprises receiving feedback in a range including at least Highly Relevant [(11) 
highly relevant], Relevant [(12) relevant], No Opinion [(13) no-opinion], Irrelevant [(14) 
non-relevant], and Highly Irrelevant [(15) highly non-relevant]" as claimed. 
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5. Claims 11-18 and 46-47 are rejected under 35 U.S.C. 102(a) as being clearly 
anticipated by the article entitled "MindReader: Querying Databases Through Multiple 
Examples" by Ishikawa et al. (hereinafter "MR") 

Referring to claim 1 1 , MR discloses a method of selecting between two types of 
matrixes as claimed. See section 3 on pages 220-221 for this disclosure. Refer 
specifically to sections 3.2 - 3.4 and Appendix D for the details of this disclosure. In 
particular, MR teaches "a method of selecting [See section 3.4] between two types of 
matrixes [full covariance matrix OR Moore-Penrose inverse matrix] to be used to weight, 
based on relevance feedback, a plurality of feature elements for image retrieval [See 
sections 1-3], the method comprising: 

selecting one of the two types of matrixes based on both a number of previously 
retrieved relevant images [N] and a length of a feature vector [n] including the plurality 
of feature elements ['the number of feedback points is less than the number of feature 
dimensions (N < n)']" as claimed. 

Referring to claim 12, MR discloses the method as claimed. See sections 3.2 - 
3.4 for this disclosure. MR's selecting one of the two types of matrixes is "based on 
both a number of previously retrieved potentially relevant images [N] which were 
identified by a user as being relevant [See Table 1 and section 3.2], and the length of 
the feature vector [n] including the plurality of feature elements [feature dimensions]" as 
claimed. 

Referring to claim 13, MR discloses the method as claimed. See sections 3.2 - 
3.4 for this disclosure. MR teaches the method as recited in claim 1 1 , as above, 
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"wherein the plurality of feature elements [feature dimensions] are all elements of the 
same feature [dimensions of the same feature]" as claimed. 

Referring to claims 14 and 15, MR discloses the method as claimed. Again see 
sections 3.2 - 3.4 for this disclosure. MR's selecting comprises using a diagonal matrix 
[Moore-Penrose inverse matrix or pseudo-inverse matrix (See Theorem 3, section 3.4 
and Appendix C - D)] if the number of retrieved relevant images [N] is less than the 
length [n] of the feature vector [N < n], and otherwise using a full matrix [M (derived from 
covariance matrix C)] as claimed. 

Claims 16 and 17 are rejected on the same basis as claims 14 and 15 
respectively, where MR's "threshold amount" is one. 

Claim 18 is rejected on the same basis as claim 11. See the discussion 
regarding claim 1 1 above for the details of this disclosure. 

Referring to claim 46, MR discloses a method of generating a query vector to 
compare to a feature vector of another image as claimed. See Section 3 of the MR 
reference for this disclosure, referring specifically to Theorem 1 and Table 1 for the 
details. In particular, MR teaches "a method of generating a query vector [q] to 
compare to a feature vector of another image, the method comprising: 

receiving feedback [goodness values v] regarding the relevance of each image 
[example image] of a set of images [set of N examples]; 

wherein N represents... [See Table 1]; and 

generating a query vector (q) corresponding to one of the plurality of features as 
follows... [See Theorem 1]" as claimed. 
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Claim 47 is rejected on the same basis as claim 46 above. See the discussion 
regarding claim 46 for the details of this disclosure. 

Claim Rejections - 35 USC § 103 

The text of those sections of Title 35, U.S. Code not included in this action can 
be found in a prior Office action. 

6. Claims 10 and 52 are rejected under 35 U.S.C. 103(a) as being unpatentable 
over RFT. 

Referring to claim 10, RFT discloses the computer readable media as recited in 
claim 1, (see above) "wherein the low-level features include: a color moments feature," 
a texture feature, and a edge [shape] feature as claimed. See the discussion of 
equations (1) - (2) on the third page of the article for this disclosure. 

RFT does not explicitly disclose that the texture feature includes "a wavelet 
based texture feature" and that the shape feature includes "a water-fill edge feature" as 
claimed. However, this is only because RFT is silent on specific examples for 
representations of texture and shape within this particular article. RFT does disclose 
that the low-level feature representations [R] can be any commonly used representation 
for a given feature f. Color histogram and color moments are used as examples of 
representations for the color feature. 

The examiner takes Official notice that wavelet based texture features were 
texture representations of common practice in the art at the time the invention was 
made, and further that water-fill edge features were shape representations of common 
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practice in the art at the time the invention was made as well. Therefore, it would have 
been obvious to one of ordinary skill in the art at the time the invention was made to use 
a wavelet based texture feature as one of the representations for texture and a water-fill 
edge feature as one of the representations for shape within RFT's system. One would 
have been motivated to do so because these common representations of the art in the 
low-level image features of texture and shape would have provided reliable 
representations of these features. 

Referring to claim 52, RFT does not explicitly disclose that the receiving 
feedback comprises receiving feedback via speech recognition as claimed. This 
however, is only because RFT is silent on the computerized means for specifying the 
user feedback. The examiner takes Official notice that the use of speech recognition for 
user input/feedback was common practice in the art at the time the invention was made. 
Therefore, it would have been obvious to one of ordinary skill in the art at the time the 
invention was made to configure RFT's system to accept the relevance feedback via 
speech recognition to obtain the invention as claimed. One would have been motivated 
to do so in order to provide the users of RFT's system with a convenient feedback 
means that was commonly used in the art. 

7. Claims 2-7, 19-30, 33-34 and 53-56 are rejected under 35 U.S.C. 103(a) as 
being unpatentable over RFT in view of MR. 

Referring to claim 2, RFT discloses the computer readable media as recited in 
claim 1 , (see above) "wherein the selecting a new set of potentially relevant images 
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[Steps 2-7 (after step 10)] comprises using a matrix [W] in determining the distance 
between one of the new plurality of query vectors and one of the plurality of feature 
vectors" as claimed. 

RFT does not explicitly disclose "dynamically selecting the matrix..." as claimed. 

MR does disclose "dynamically selecting the matrix [M in this case] based on 
both a number of images [N] in the set of potentially relevant images for which 
relevance feedback was input and a number [n] of feature elements in the one feature 
vector" as claimed. See the discussion regarding claim 11 above. 

It would have been obvious to one of ordinary skill in the art at the time the 
invention was made to add MR's functionality of dynamically selecting the weighting 
matrix to the system and method of RFT to obtain the invention as claimed. One would 
have been motivated to do so in order to avoid the problem of a singular and non- 
invertible covariance matrix, as disclosed by MR. One would have been further 
motivated to combine the teachings of MR and RFT because MR explicitly builds off of 
the MARS system [MR: See sections 2 - 3], which is the system described in the RFT 
reference [RFT: See the first paragraph of section 4]. Thus, the MR reference directly 
suggests this combination. 

Referring to claim 3, the system and method of RFT in view of MR as applied to 
claim 2 above discloses the invention as claimed. In particular, MR (as added to RFT) 
teaches the dynamically selecting exactly as claimed. See the discussions of claims 14 
and 15 above for the details of this disclosure. 
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Referring to claim 4, the system and method of RFT in view of MR as applied to 
claim 2 above discloses the invention as claimed. In particular, MR (as added to RFT) 
teaches the dynamically selecting comprising: 

"if the number of images [N] in the set of potentially relevant images for which 
relevance feedback was input is not less than [N >= n] the number of feature elements 
in the one feature vector [n], then using one matrix [covariance matrix C] that transforms 
the query vector and the one feature vector to a higher-level feature space and then 
using another matrix [M] that assigns a weight to each element of the transformed query 
vector and the transformed feature vector; and 

if the number of images [N] in the set of potentially relevant images is less than 
[N < n] the number of feature elements in the one feature vector [n], then using a matrix 
[Moore-Penrose inverse matrix or pseudo-inverse matrix] that assigns a weight to each 
element of the query vector and then one feature vector [See Appendix D]" as claimed. 

Referring to claim 5, the system and method of RFT in view of MR as applied to 
claim 2 above discloses the invention as claimed. See Theorem 2 on page 221 of the 
MR reference for the details of this disclosure. MR's dynamic matrix weighting, as 
added to the system and method of RFT, discloses the generation of the matrix W* [M 
in the MR reference] as claimed. 

Referring to claim 6, the system and method of RFT in view of MR as applied to 
claim 2 above discloses the invention as claimed. See Theorem 3 on page 221 of the 
MR reference for the details of this disclosure. MR's dynamic matrix weighting, as 
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added to the system and method of RFT, discloses the generation of the diagonal 
matrix elements w kk [rrijj in the MR reference] as claimed. 

Referring to claim 7, RFT does not explicitly disclose the details of the generation 
of a new query vector as claimed because RFT is silent on these details. However, MR 
does disclose the generation of new query vectors exactly as claimed. See Theorem 1 
on page 221 of the MR reference for the details of this disclosure. It would have been 
obvious to one of ordinary skill in the art at the time the invention was made to integrate 
MR's method of generating the new query vectors into the system and method of RFT. 
One would have been motivated to do so for the same reasons as discussed above with 
regard to claim 2. 

Claims 19 and 20 are rejected on the same basis as claim 2. See the 
discussions regarding claims 1 and 2 above for the details of this disclosure. 

Claims 21 and 22 are rejected on the same basis as claims 14 and 15 
respectively, in light of the basis for claim 19. See the discussions regarding claims 1 , 2 
and 14-15 above for the details of this disclosure. 

Claims 23, 24 and 29-30 are rejected on the same basis as claim 4. See the 
discussions regarding claims 1, 2 and 4 above for the details of this disclosure. 

Referring to claim 25, the system and method of RFT in view of MR as applied to 
claim 2 above discloses the invention as claimed. See the discussions regarding claims 
1 , 2 and 4 for this disclosure. In particular, MR's dynamic matrix weighting, as added to 
the system and method of RFT, discloses the determination of the distance as claimed. 
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See equation (4) on page 221 of the MR reference for this disclosure. MR's equation 
(4) is equivalent to the claimed equation. 

Referring to claim 26, the system and method of RFT in view of MR as applied to 
claim 2 above discloses the invention as claimed. See the discussions regarding claims 
1 , 2 and 4 for this disclosure. In particular, MR's dynamic matrix weighting, as added to 
the system and method of RFT, discloses the determination of the distance as claimed. 
See equation (4) on page 221 of the MR reference in light of the disclosure of section 
3.4 and Appendix D. MR's equation (4), as modified by Appendix D in the case where 
N < n, is equivalent to the claimed equation. 

Referring to claims 27 and 28, the system and method of RFT in view of MR as 
applied to claim 2 above discloses the invention as claimed. See the discussions 
regarding claims 1 , 2 and 4 for this disclosure. In particular, the method of RFT (as 
modified by MR) is repeated for each of the plurality of features as claimed. See Steps 
4 - 6 of RFT's method for this disclosure. 

Claims 33-34 are rejected on the same basis as claim 4, in light of the basis for 
claim 32 above. See the discussions regarding claims 1,2,4 and 32 above for the 
details of this disclosure. 

Claims 53 and 54 are rejected on the same basis as claims 51 and 52 
respectively, in light of the basis for claim 19. See the discussions regarding claims 1, 
19 and 51-52 above for the details of this disclosure. 

Claim 55 is rejected on the same basis as claims 7 and 51 above. See the 
discussions regarding claims 1 , 7 and 51 above for the details of this disclosure. 
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Claim 56 is rejected on the same basis as claim 52 above, in light of the basis for 
claim 55. See the discussions regarding claims 1, 7 and 51-52 above for the details of 
this disclosure. 

Response to Arguments 

8. Applicant's arguments with respect to claims 1-30, 32-38, 40-44 and 46-56 have 
been considered but are moot in view of the new ground(s) of rejection. 

Conclusion 

9. The prior art made of record and not relied upon is considered pertinent to 
applicant's disclosure. 

U.S. Patent No. 6,345,274 to Zhu et al. and the article entitled "Using Relevance 
Feedback in Content-Based Image Metasearch" by Benitez et al. are both considered 
particularly pertinent to applicant's disclosure and claimed invention. 

1 0. Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to Brian Goddard whose telephone number is 703-305- 
7821. The examiner can normally be reached on M-F, 9 AM - 5 PM. 

If attempts to reach the examiner by telephone are unsuccessful, the examiner's 
supervisor, Safet Metjahic can be reached on 703-308-1436. The fax phone numbers 
for the organization where this application or proceeding is assigned are 703-746-7239 
for regular communications and 703-746-7238 for After Final communications. 
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Any inquiry of a general nature or relating to the status of this application or 
proceeding should be directed to the receptionist whose telephone number is 703-305- 
3900. 

bdg 

April 30, 2003 




